Multiple comparisons tests

function correlationMcTest(x::UniData, Y::UniDataVec;
            direction::TestDir = Both(),
            switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)), 
            nperm::Int = 20_000, 
            seed::Int = 1234, 
            verbose::Bool = true, 
            #
            standardized::Bool = false,
            #
            stepdown::Bool = true,
            fwe::Float64 = 0.05,
            threaded::Bool = Threads.nthreads()>=4) where TestDir <: TestDirection

Multiple comparisons Pearson product-moment correlation test by data permutation.

Run $M$ correlation tests simultaneously. The input data are a fixed vector x and $M$ vectors given as Y, a vector of $M$ vectors $y_1,...,y_M$.

x and all vectors in Y must have equal length.

The $M$ null hypotheses have form

$H_0(m):r_{(x, y_m)}=0, \quad m=1...M$,

where $r_{x,y_m}$ is the Pearson correlation coefficient between vector x and the $m^{th}$ vector of Y.

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded, see here.

If standardized is true, both x and all vectors of Y are assumed standardized (zero mean and unit standard deviation). With standardized input data the test can be executed faster as in this case the cross-product is actually the correlation. If standardized is false, the data will be standardized to execute a faster test.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version correlationTest

Aliases: rMcTest, trendMcTest

Return a MultcompTest structure.

Examples

using PermutationTests
N=10; # number of observations
M=100; # number of tests
x=randn(N);
Y=[randn(N) for i=1:M];
t=rMcTest(x, Y) # bi-directional test

tR=rMcTest(x, Y; direction=Right()) # right-directional test
tL=McTest(x, Y; direction=Left()) # left-directional test
tMC=rMcTest(x, Y; switch2rand=1) # Force a monte carlo test
# Force a monte carlo test and performs 50K permutations
t5K=rMcTest(x, Y; switch2rand=1, nperm= 50_000) 
tnoSD=rMcTest(x, Y; stepdown=false) # don't do stepdown
tnoMT=rMcTest(x, Y; threaded=false) # don't run it multithreaded
t001=rMcTest(x, Y; fwe=0.01) # stepdown rejects at 0.01 level instead of 0.05

Similar tests

See correlationTest

function correlationMcTest!(<same args and kwargs as `correlationMcTest`>)

As correlationMcTest, but x is overwritten if standardized is true.

Aliases: rMcTest!, trendMcTest!

Univariate version: correlationTest!

function trendMcTest(<same args and kwargs as correlationMcTest>)

function trendMcTest!(<same args and kwargs as correlationMcTest!>)

Actually aliases of correlationMcTest and correlationMcTest!, respectively.

x is any specified trend (linear, polynomial, exponential, logarithmic, trigonometric, ...) and Y holds the observed data. A multiple comparison Pearson product-moment correlation test between x and all $M$ vectors in Y is then carried out.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate versions trendTest or trendTest!

Both methods return a MultcompTest structure.

Examples

using PermutationTests
# We are goint to test an upward linear trend
N=10
M=100
x=Float64.(collect(Base.OneTo(N))) # [1, 2,..., N]
# out of the M vectors created here below, only one has a significant correlation
Y=[[1., 2., 4., 3., 5., 6., 7., 8., 10., 9.], ([randn(N) for m=1:M-1]...)];
# Since we expect an upward linear trend, the correlation is expected to be positive,
# hence we use a right-directional test to increase the power of the test.
t = trendMcTest(x, Y; direction=Right()) 

function pointBiSerialMcTest(<same args and kwargs as `studentMcTestIS`>)

Actually an alias for studentMcTestIS.

Run $M$ point bi-serial correlation tests simultaneously. The correlations are between the $M$ input data vectors $y_1,...,y_M$ given as argument Y, all holding $N=N_1+N_2$ elements, and a vector $x$, internally created, with the first $N_1$ elements equal to 1 and the remaining $N_2$ elements equal to 2.

If you need to use other values for the dicothomous variable $x$ or a different order for its elements, use correlationMcTest instead.

The $M$ null hypotheses have form

$H_0(m): b_{(x,y_m)}=0, \quad m=1...M$,

where $b_{(x,y_m)}$ is the point bi-serial correlation between $y_m$ (the $m^{th}$ input data vectors in Y) and the internally created vector $x$.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version pointBiSerialTest

Return a MultcompTest structure.

Examples

using PermutationTests
ns=[4, 6]; # N1=4, N2=6
N=sum(ns); # number of observations

Y = [rand(N) for m=1:M]; # some Gaussian data as example
# implicitly, the point bi serial correlation is between 
# y1,...,yM and x=[1, 1, 1, 1, 2, 2, 2, 2, 2, 2]
t=pointBiSerialMCTest(Y, ns) # by default the test is bi-directional

tR=pointBiSerialMCTest(Y, ns; direction=Right()) # right-directional test
tL=pointBiSerialMCTest(Y, ns; direction=Left()) # left-directional test

# METHOD (1)
function studentMcTestIS(Y::UniDataVec, ns::IntVec;
                direction::TestDir = Both(),
                switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)),
                nperm::Int = 20_000, 
                seed::Int = 1234, 
                verbose::Bool = true,
                #
                stepdown::Bool = true,
                fwe::Float64 = 0.05,
                threaded::Bool = Threads.nthreads()>=4,
                asPearson::Bool = true) where TestDir <: TestDirection

# METHOD (2)
function studentMcTestIS(Yvec::UniDataVec²; <same kwargs>)

# METHOD (3)
function studentMcTestIS(X::UniDataVec, Y::UniDataVec; <same kwargs>)

METHOD (1)

Multiple comparisons Student's t-test for independent samples by data permutation.

Run $M$ t-tests simultaneously. Given $M$ hypotheses with $N=N_1+N_2$ observations for two groups each, the $M$ null hypotheses have form

$H_0(m): μ_{m1}=μ_{m2}, \quad m=1...M$,

where $μ_{m1}$ and $μ_{m2}$ are the mean of group 1 and group 2, respectively, for the $m^{th}$hypothesis.

For a bi-directional test, this t-test is equivalent to the 1-way ANOVA for two independent samples. However, in contrast to the ANOVA, it can be directional.

ns is a vector of integers holding the group numerosity $N_1, N_2$ (see examples below).

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

If asPearson is true(default), the test is run as an equivalent version of a Pearson correlation test. This is in general advantageous for multiple comparison tests, especially if approximate (see the benchmarks). If you seek best performance for exact tests, benchmark the speed of the test with asPearson set to true and to false to see what version is faster for your data.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version studentTestIS

Aliases: tTestMcIS, pointBiSerialMcTest

Return a MultcompTest structure.

METHOD (2)

As method (1), but input data Yvec is a vector containing $M$ pairs of vector of arbitrary length, holding in the natural order the data corresponding to the $m^{th}$ hypothesis for group 1 and group 2, respectively.

METHOD (3)

As method (1), but input data X and Y holds $M$ vectors of observations each, X corresponding to data for group 1 and Y corresponding to data for group 2. The $M$ vectors in X must have all the same length ($N_1$), so must the $M$ vectors in Y ($N_2$).

Examples

# (1)
using PermutationTests
M=100 # number of hypotheses
ns=[4, 5] # number of observations in group 1 and group 2 (N1 and N2)
N=sum(ns) # total number of observations
Yvec = [[randn(n) for n in ns] for m=1:M]; # some random Gaussian data for example 
Y=[vcat(yvec...) for yvec in Yvec];
t = tMcTestIS(Y, ns) # by default the test is bi-directional

# Force an approximate test with 10000 random permutations
tapprox = tMcTestIS(Y, ns; switch2rand=1, nperm=10000) 
tR=tMcTestIS(Y, ns; direction=Right()) # right-directional test
tL=tMcTestIS(Y, ns; direction=Left()) # left-directional test

# with a bi-directional test, t is equivalent to a 1-way ANOVA for independent samples
tanova= fMcTestIS(Y, ns) 
println(sum(abs.(t.p - tanova.p)) ≈ 0. ? "OK" : "error")

# do not run it using the CrossProd test statistic
tcor = tMcTestIS(Y, ns; asPearson=false) 

# in method (2) only the way the input data is formatted is different 
t2 = tMcTestIS(Yvec)
println(sum(abs.(t.p - t2.p)) ≈ 0. ? "OK" : "error")

# in method (3) also, only the way the input data is formatted is different 
t3 = tMcTestIS([Yvec[m][1] for m=1:M], [Yvec[m][2] for m=1:M])
println(sum(abs.(t.p - t3.p)) ≈ 0. ? "OK" : "error")

Similar tests

See studentTest1S

# METHOD (1)
function anovaMcTestIS(Y::UniDataVec, ns::IntVec;
                direction::TestDir = Both(),
                switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)),
                nperm::Int = 20_000, 
                seed::Int = 1234, 
                verbose::Bool = true,
                #
                stepdown::Bool = true,
                fwe::Float64 = 0.05,
                threaded::Bool = Threads.nthreads()>=4) where TestDir <: TestDirection

# METHOD (2)
function anovaMcTestIS(𝐘vec::UniDataVec²; <same kwargs>)

METHOD (1)

Multiple comparisons 1-way analysis of variance (ANOVA) for independent samples by data permutation.

Run $M$ ANOVAs simultaneously. The Input data is given as a vector Y holding $M$ vectors $𝐲1,...,𝐲M$ concatenating all observations, that is, holding each $N=N_1+...+N_K$ observations for $K>2$ independent samples (groups). The observations are ordered with group 1 first, then group 2,..., finally group K. Note that the group numerosity $N_1,...,N_K$ must be the same for all $M$ hypotheses. The only check performed is that the first vector in Y contains sum(ns) elements.

The $M$ null hypotheses have form

$H_0(m): μ_{m1}= \ldots =μ_{mK}, \quad m=1...M$,

where $μ_{mk}$ is the mean of the $k^{th}$ group for the $m^{th}$ hypothesis.

ns is a vector of integers holding the group numerosity $N_1,...,N_K$ (see examples below).

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version anovaTestIS

Alias: fMcTestIS

Return a MultcompTest structure.

METHOD (2)

As method (1), but input data Yvec is a vector holding $M$ vectors of K vectors of observations for group 1,..., group K.

Examples

# method (1)
using PermutationTests
ns=[3, 4, 5] # number of observations in group 1, 2 and 3
M=10 # number of hypotheses
Yvec = [[randn(n) for n in ns] for m=1:M]; # some random Gaussian data for example 
t = fMcTestIS([vcat(y...) for y in Yvec], ns) # ANOVA tests are always bi-directional

# Force an approximate test with 5000 random permutations
tapprox = fMcTestIS([vcat(y...) for y in Yvec], ns; switch2rand=1, nperm=5000) 

# in method (2) only the way the input data is formatted is different 
t2 = fMcTestIS(Yvec)
# of course, method (1) and (2) give the same p-values
println(sum(abs.(t.p-t2.p))≈0. ? "OK" : "error") 

Similar tests

See anovaTestIS

function chiSquaredMcTest(tables::AbstractVector{Matrix{I}};
            direction::TestDir = Both(),
            switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)), 
            nperm::Int = 20_000, 
            seed::Int = 1234, 
            verbose::Bool = true,            
            #
            stepdown::Bool = true,
            fwe::Float64 = 0.05,
            threaded::Bool = Threads.nthreads()>=4,
            asPearson::Bool = true) 
                    where {I <: Int, TestDir <: TestDirection}

Multiple comparisons chi-squared ($\chi^2$) permutation test for $2 \cdot K$ contingency tables, where $K$ is ≥2. It tests simultaneously $M$ contingency tables, which must all have same dimension and same column sums.

The $M$ null hypotheses have form

$H_0(m): O_m=E_m$, \quad m=1...M``,

where $O_m$ and $E_m$ are the observed and expected frequencies of the $m^{th}$contingency table.

tables is a vector of $M$ contingency tables. See chiSquaredTest for more explanations.

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

For $K=2$ this function calls studentMcTestIS and pass to it also argument asPearson, otherwise calls anovaMcTestIS and argument asPearson is ignored.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version chiSquaredTest

Aliases: Χ²McTest, fisherExactMcTest

Return a [MultcompTest] structure.

Examples

using PermutationTests
tables=[[3 3 2; 0 0 1], [1 2 2; 2 1 1], [3 1 2; 0 2 1]];
t=Χ²McTest(tables) # the test is bi-directional

tables=[[6 1; 2 5], [4 1; 4 5], [5 2; 3 4]]
tR=fisherExactMcTest(tables; direction=Right()) 
# or tR=Χ²Test(tables; direction=Right())

# do not use PearsonR statistic
tR_=fisherExactMcTest(tables; direction=Right(), asPearson=false) 

function fisherExactMcTest(<same args and kwargs as `chiSquaredMcTest`>)

Actually an alias for chiSquaredMcTest. It can be used for 2x2 contingency tables. See chiSquaredTest for more explanations.

Univariate version: fisherExactTest

Return a MultcompTest structure.

Examples

using PermutationTests
tables=[[6 1; 2 5], [4 1; 4 5], [5 2; 3 4]];
tR=fisherExactMcTest(tables; direction=Right()) 
# or tR=Χ²Test(tables; direction=Right())

function studentMcTestRM(<same args and kwargs as `studentMcTest1S`>)

Actually an alias for studentMcTest1S

In order to run a multiple comparisons t-test for repeated measure, use as data input the vector of $M$ vectors of differences across the two measurements (or treatments, time, etc.).

Do not change the refmean default value. See studentMcTest1S for more details.

Directional tests, permutation scheme and number of permutations for exact tests: as per studentTest1S

Univariate version: studentTestRM

Alias: tMcTestRM

Return a MultcompTest structure.

Examples

using PermutationTests
N=10 # number of observation per treatment
M=100 # number of hypotheses

# suppose you have data as
Y1=[randn(N) for m=1:M]; # measurement 1
Y2=[randn(N) for m=1:M]; # measurement 2

# Let us compute the differences as
Y=[y1-y2 for (y1, y2) in zip(Y1, Y2)];
t=tMcTestRM(Y) # by default the test is bi-directional

tR=tMcTestRM(Y; direction=Both()) #  right-directional test
# if test tR is significant for some hypotheses, 
# for these hypotheses the mean of measurement 1 
# exceeds the mean of measurement 2.

(2) function studentMcTestRM!(<same args and kwargs as studentMcTest1S!>)

Actually an alias for studentMcTest1S!.

Y is overwritten in the case of approximate (random permutations) tests.

Alias: tMcTestRM!

# METHOD (1)
function anovaMcTestRM(Y::UniDataVec, ns::@NamedTuple{n::Int, k::Int};
            direction::TestDir = Both(),
            switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)),
            nperm::Int = 20_000, 
            seed::Int = 1234, 
            verbose::Bool = true,
            #
            stepdown::Bool = true,
            fwe::Float64 = 0.05,
            threaded::Bool = Threads.nthreads()>=4) where TestDir <: TestDirection

# METHOD (2)
function anovaMcTestRM(Yvec::UniDataVec²; <same kwargs>)

METHOD (1)

Multiple comparison 1-way analysis of variance (ANOVA) for repeated measures by data permutation. Given $M$ hypotheses, each with $K$ repeated measures (e.g., treatments, time, etc.) for each of $N$ observation units (e.g., subjects, blocks, etc.), the null hypotheses have form

$H_0(m): μ_{m1}= \ldots =μ_{mk}, \quad m=1...M$,

where $μ_{mk}$ is the mean of the $k^{th}$ treatment for the $m^{th}$ hypothesis.

Y is a vector hoding $M$ vectors, each one concatenaning the $K$ treatments (treatment 1,..., treatment $K$) for each observation for the $m^{th}$ hypothesis in this order: the $K$ treatments for observation 1, the $K$ treatments for observation 2, ..., the $K$ treatments for observation $N$. Thus, Y holds $M$ vectors of $N \cdot K$ elements.

ns is a julia named tuple with form (n=N, k=K) (see examples below).

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version anovaTestRM

Alias: fTestMcRM

Return a MultcompTest structure.

METHOD (2)

As (1), but Yvec is a vector of $M$ vectors, each one holding $N$ vectors holding each the $K$ treatments for the $n^{th}$ subject (see examples below).

Examples

# method (1)
using PermutationTests
N=6 # number of observation units per treatment
K=3 # number of treatments
M=10 # number of hypotheses
Yvec = [[randn(K) for n=1:N] for m=1:M]; # some random Gaussian data for example 
t = fMcTestRM([vcat(yvec...) for yvec in Yvec], (n=N, k=K)) 
# ANOVA tests are always bi-directional

# Force an approximate test with 5000 random permutations
tapprox = fMcTestRM([vcat(yvec...) for yvec in Yvec], (n=N, k=K); 
            switch2rand=1, nperm=5000)

# in method (2) only the way the input data is formatted is different 
 
t2 = fMcTestRM(Yvec)
println(sum(abs.(t.p - t2.p)) ≈ 0. ? "OK" : "error")
println(sum(abs.(t.obsstat - t2.obsstat)) ≈ 0. ? "OK" : "error")

Similar tests

See anovaTestRM

function cochranqMcTest(tables::AbstractVector{Matrix{I}};
            direction::TestDir = Both(),
            switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)),
            nperm::Int = 20_000, 
            seed::Int = 1234, 
            verbose::Bool = true,
            #
            stepdown::Bool = true,
            fwe::Float64 = 0.05,
            threaded::Bool = Threads.nthreads()>=4) 
                where {I <: Int, TestDir <: TestDirection}

Multiple comparisons Cochran Q by data permutation.

The Cochran Q test is analogous to the 1-way ANOVA for repeated measures, but takes as input dicothomous data (zeros and ones). Given $M$ hypotheses, consisting each in $N$ observation units (e.g., subjects, blocks, etc.) and $K$ repeated measures (e.g., treatments, time, etc.), the null hypotheses have form

$H_0(m): μ_{m1}= \ldots =μ_{mk}, \quad m=1...M$,

where $μ_{mk}$ is the mean of the $k^{th}$ treatment.

Input tables is a vector of $M$ tables of zeros and ones with size $NxK$, where $N$ is the number of observations and $K$ the repeated measures. See cochranqTest for more explanations.

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

Directional tests, Permutation scheme and number of permutations for exact tests: as per univariate version cochranqTest

Aliases: qMcTest, mcNemarMcTest

Return a MultcompTest structure.

Examples

using PermutationTests
tables=[ [1 1 0; 1 0 0; 1 1 1; 1 1 0; 1 0 1; 1 1 0],
        [1 0 0; 1 1 0; 1 1 0; 1 1 1; 1 0 0; 1 0 0],
        [1 0 0; 0 0 1; 1 0 1; 1 1 0; 1 0 1; 1 0 0]];
t=qMcTest(tables) # the test with K>2 can only be bi-directional

function mcNemarMcTest(same args and kwargs as `cochranqMcTest`>)

Actually an alias for [cochranqMcTest)(@ref).

Run $M$ McNemar test simultaneously.

Input tables is a vector of $M$ tables of zeros and ones with size $Nx2$, where $N$ is the number of observations and $2$ the number of repeated measures. See cochranqTest for more explanations.

Univariate version: mcNemarTest

Return a MultcompTest structure.

Examples

using PermutationTests
tables=[[1 1; 1 0; 1 0; 0 0; 1 0; 1 0],
        [1 0; 1 1; 1 0; 0 1; 0 0; 1 0],
        [0 1; 0 0; 1 0; 1 0; 1 0; 1 1]];
t=mcNemarMcTest(tables) # by default the test is bi-directional

tR=mcNemarMcTest(tables; direction=Right()) # right-directional test

function studentMcTest1S(Y::UniDataVec;
            refmean::Realo = nothing,
            direction::TestDir = Both(),
            switch2rand::Int = max(Int(1e8) ÷ length(𝐘), Int(1e4)),
            nperm::Int = 20_000, 
            seed::Int = 1234, 
            verbose::Bool = true,
            #
            stepdown::Bool = true,
            fwe::Float64 = 0.05,
            threaded::Bool = Threads.nthreads()>=4) where TestDir <: TestDirection 

Multiple comparison one-sample t-test by data permutation.

Run $M$ one-sample t-tests simultaneously. The null hypotheses have form

$H_0(m): μ_m=μ_0, \quad m=1...M$,

where $μ_m$ is the mean of the observations for the $m^{th}$ hypothesis and $μ_{0}$ is a reference population mean.

refmean is the reference mean ($μ_0$) above. The default is 0.0, which is the value needed in most situations.

Y is a vector of $M$ vectors holding each the $N$ observations which mean is to be compared to $μ_0$.

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version studentTest1S

Alias: tTest1S

Return a MultcompTest structure.

Examples

using PermutationTests
N=20 # number of observations
M=100 # number of hypotheses
y = [randn(N) for m=1:M]; # some random Gaussian data for example
t = tMcTest1S(Y) # By deafult the test is bi-directional

tR = tMcTest1S(Y; direction=Right()) # right-directional test
tL = tMcTest1S(Y; direction=Left()) # Left-directional test

# Force an approximate test with 5000 random permutations
tapprox = tMcTest1S(Y; switch2rand=1, nperm=5000) 

# test H0m: μ(ym)=1.5: all will be rejected as the expected mean is 0.0
t1 = tMcTest1S(Y; refmean=1.5) 

Similar tests

See studentTest1S

function studentMcTest1S!(<same args and kwargs as `studentMcTest1S`>)

As studentMcTest1S, but Y is overwritten in the case of approximate (random permutations) tests.

Alias: tTest1S!

Univariate version: studentTest1S!

function signMcTest(Y::Union{AbstractVector{BitVector}, AbstractVector{Vector{Bool}}};
            direction::TestDir = Both(),
            switch2rand::Int= max(Int(1e8) ÷ length(𝐘), Int(1e4)),
            nperm::Int = 20_000, 
            seed::Int = 1234, 
            verbose::Bool = true,
            #
            stepdown::Bool = true,
            fwe::Float64 = 0.05,
            threaded::Bool = Threads.nthreads()>=4) where TestDir <: TestDirection

Multiple comparisons sign test by data permutation.

Run $M$ sign tests simultaneously.The null hypotheses have form

$H_0(m): E_m(true)=E_m(false), \quad m=1...M$,

where $E_m(true)$ and $E_m(false)$ are the expected number of true and false occurrences, respectively, in the $m^{th}$ hypothesis.

Y ia a vector of $M$ vectors holding each $N$ booleans.

For optional keyword arguments, direction, switch2rand, nperm, seed, verbose, stepdown, fwe and threaded see here.

Directional tests, permutation scheme and number of permutations for exact tests: as per univariate version signTest

Return a MultcompTest structure.

Examples

using PermutationTests
N=20; # number of observations
M=100; # number of hypotheses
Y = [rand(Bool, N) for m=1:M]; # some random Gaussian data for example
t = signMcTest(Y) # By deafult the test is bi-directional

tR = signMcTest(Y; direction=Right()) # right-directional test
tL = signMcTest(Y; direction=Left()) # Left-directional test

# Force an approximate test with 5000 random permutations
tapprox = signMcTest(Y; switch2rand=1, nperm=5000)